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Abstract 


1 

Work  in  progress  on  JETS,  the  successor  to  PLANES,  is  described.  JETS  is 
a  natural  language  question  answering  system  that  is  intended  to  interface 
users  to  a  large  relational  data  base.  The  architecture  is  designed  to  extend 
the  conceptual  coverage  of  JETS  to  better  meet  the  conversational  *r.d  data 
base  usage  requirements  of  users.  The  implementation  of  JETS  is  designed  to 
gain  a  high  degree  of  closure  over  concept  taanipulation,  contributing  to  a 
solution  to  the  problems  of  perspicuity  and  scale.  Specific  examples  are 
given  of  concept  manipulation  through  the  implied  relationships  of 
modification  and  of  an  approach  to  problem-solving  through  the  use  of  frames. 
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Achieving  Completeness  Through  Coversge  and  Closure 

JETS  is  a  natural  language  question  answering  system  that  is  currently 
under  development.  Its  goals  and  design  arc  an  outgrowth  of  the  experience 
gained  in  the  development  and  user  environment  testing  of  PLANES  [Valt*  and 
Goodman,  1977]  [Walts,  1978].  This  paper  will  describe  JETS  in  terms  of 
motivation,  performance  goals,  strategies  for  achieving  the  goals,  and 
techniques  for  measuring  performance. 

PLANES  is  a  system  that  was  developed  to  study  the  problems  of  natural 
language  access  to  a  very  large  data  base.  The  data  base  which  was  used 
contains  naval  aircraft  maintenance  and  flight  records,  and  it  is  the  same 
data  base  that  JETS  will  be  applied  to.  The  thrust  of  PLANES  was  toward  the 
investigation  of  natural  language  data  base  interfaces  that  could  be 
engineered  to  comply  with  response  time  and  machine  size  constraints  while 
enabling  the  user  to  express  questions  in  the  language  to  which  he  is 
accustomed.  PLANES  accepts  user  utterances  that  may  or  may  not  comply  with 
standard  grammar,  that  may  include  pronominal  reference,  and  that  may  include 
some  forms  of  ellipsis.  In  an  effort  to  make  an  objective  assessment  of  the 
achievements  of  PLANES,  testing  was  conducted  using  the  techniques  described 
in  [Tennant,  1979].  The  testing  revealed  that  in  addition  to  problems  of 
linguistic  completeness,  there  are  interesting  questions  about  the  conceptual 
completeness  of  question  answering  systems.  JETS  is  the  response  to  those 
questions. 

Coverage  .C0fflPl<?teHfi.aa 

The  performance  of  question  answering  systems  can  be  thought  of  as  having 
two  nearly  independent  components:  the  elements  of  conceptual  coverage  and  the 
elements  of  linguistic  coverage.  The  conceptual  coverage  of  a  system  refers 
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to  the  set  of  concepts  that  it  can  deal  with.  The  linguistic  coverage  of  a 
system  refers  to  the  set  of  linguistic  features  that  have  been  included  in  it 
to  allow  for  variations  in  the  way  the  concepts  are  referenced.  Syntactic 
structure.,  anaphoric  reference  and  ellipsis  are  all  examples  of  elements  of 
linguistic  coverage.  Conceptual  coverage  and  linguistic  coverage  are 
attributes  of  the  program.  The  adequacy  of  the  conceptual  and  linguistic 
coverage  may  be  measured  against  the  demands  that  a  set  of  users  place  on  the 
system.  The  measures  are  called  conceptual  and  linguistic  completeness, 
respectively.  The  attributes,  conceptual  and  linguistic  coverage,  a  d  the 
measurements,  conceptual  and  linguistic  completeness,  are  described  in  roore 
detail  in  iTennant,  19793. 

The  major  work  on  PLANES,  as  on  most  question  answering  systems,  has  been 
to  provide  the  user  with  a  habitable  [Watt,  1968]  system  through  high 
linguistic  completeness.  Our  testing  shows  that  linguistic  completeness  is 
not  the  only  limiting  factor  in  building  habitable  natural  language  systems. 
The  users  wanted  to  be  able  to  refer  to  certain  concepts,  but  were  unable  to 
do  so.  For  example,  users  often  asked  questions  about  the  structure  of  the 
data  base  or  asked  for  definitions.  They  made  statements  to  set  the  context 
for  later  questions,  or  attempted  to  summarise  a  portion  of  the  dialogue 
expecting  agreement  or  disagreement  with  their  summaries.  They  made 
presuppositions  about  the  data  that  were  not  justified.  In  short,  there  were 
a  significant  number  of  concepts  that  users  attempted  to  reference  or 
capabilities  that  they  attempted  to  use  that  were  not  included  in  PLANES, 

In  addition  to  the  voids  in  conceptual  coverage  chat  were  discovered 
while  testing  PLANES,  it  was  observed  that  the  users  readily  adapted  to  the 
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limitations  of  PLANES'  linguistic  coverage.  The  user's  "intellectual  overhead" 
or  "distraction  from  the  problem"  was  not  measurable,  but  they  did  appear  to 
be  able  to  express  themselves.  We  feel  that  improving  the  linguistic 
completeness  of  question  answerers  is  indeed  an  area  where  attention  should  ;  i  ; 
continue  to  be  focused.  However,  linguistic  completeness  is  only  part,  of  the 
problem  of  building  habitable  natural  language  systems.  The  work  on  JETS  is  - 
centered  on  improving  the  conceptual  completeness  of  natural  language  question 
answering  systems. 

With  this  motivation,  our  work  on  JETS  has  four  goals.  First,  we  intend 
to  enlarge  the  domain  of  discourse.  Instead  of  simply  enabling  a  question 
answerer  to  accept  questions  about  a  data  base,  it  must  aiso  be  able  to  handle 
references  to  objects  defined  in  the  discourse.  This  includes  characterizing 
the  responses  to  queries  that  it  gsnerates  to  resolve  references  ;c  its  own 
utterances  by  the  users.  It  should  be  able  to  handle  references  like  "the 
last  plane  I  mentioned"  and  "the  last  question".  It  should  be  able  to  clarify 
for  the  user  how  it  interprets  vague,  ambiguous,  or  technical  phrases. 

Second,  the  natural  language  component  should  take  on  greater 
responsibility  for  pragmatic  knowledge.  It  should  have  two  distinct 
components.  One  component  contains  knowledge  about  the  data  in  the  data  base, 
apecifically  including  relationships  between  data  types  and  between  individual 
data  elementa.  The  other  component  contains  knowledge  about  the  activities 
that  are  reported  on  in  the  data  base,  rather  than  just  the  data  in  the  data 
base.  We  have  seen  users  refer  to  concepts  that  are  related  to  the 
activities,  but  that  are  not  in  the  data  base.  A  natural  language  system 
should  be  able  to  understand  these  references  if  only  to  be  more  helpful  in 
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explaining  why  the  user's  question  cannot  be  answered.  The  following 
responses  could  be  generated  by  JETS  only  if  references  to  concepts  outside 
the  range  of  the  data  base  were  understood: 


1.  There  is  no  data  on  individual  pilots  in  the 
data  base. 

2.  Urcraft  are  not  grouped  by  squadron  in  the 
data  base,  but  they  can  be  grouped  by  permanent 
unit  assignments. 

3.  The  data  base  does  not  describe  all  combat 
airoraft,  only  fighter  and  attack  aircraft. 

4.  The  data  base  ac.*3  not  specify  the  part 
that  failed,  only  the  system  or  subsystem 
that  contained  the  failure. 


The  third  goal  fer  JETS  is  to  break  away  from  the  familiar  model  for 
question  answerers  of  interpreting  each  user  utterance  as  a  data  base  query. 

have  observed  that  users  dr  not  intend  each  utterance  as  a  query.  Some  set 
the  context  for  later  discourse  or  give  general  instructions  suoh  as  data 
formatting.  Some  take  several  sentences  to  specify  one  query,  while  others 
couch  several  queries  in  one  sentence.  JETS  builds  an  internal  description  of 
the  user's  utterance,  and  takes  action  on  it  depending  on  the  user  demands 
that  are  implied  in  the  utterance. 

Our  fourth  goal  is  to  evaluate  the  performance  of  JETS  with  respect  to 
its  conceptual  completeness.  Designing  and  testing  explicitly  for  linguistic 
completeness  is  a  goal  for  future  research.  Our  conceptual  completeness  tests 
will  consist  of  giving  short  descriptions  of  the  domain  of  discourse  to 
subjects,  each  acting  in  two  roles.  First,  acting  as  composers,  they  will 
generate  data  base  problems.  Then,  acting  as  users,  they  will  try  to  solve 
problems  generated  by  the  other  subjects.  By  having  subjects  who  are  only 
slightly  familiar  with  the  data  base  compose  problems,  the  problems  will  not 
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be  biased  toward  those  that  can  actually  be  solved  using  the  data  base,  this 
is  intended  to  simulate  the  data  base  problems  that  would  occur  to  a  casual 
user,  one  who  is  not  very  familiar  with  the  exact  contents  of  the  data  base. 

the  users  will  interact  with  JEtS  through  a  human  interpreter,  whose  task 
will  be  tc  maintain  the  conceptual  content  of  the  user  utterances  while  making 
them  oomprehendable  to  JEtS.  In  other  words,  the  interpreter  compensates  for 
potential  deficiencies  in  linguistic  coverage,  the  conceptual  completeness  of 
the  system  can  be  measured  by  computing  the  fraction  of  user  utterances  that 
were  properly  handled  by  JEtS. 

Claaurg 

the  architecture  of  JEtS  centers  on  establishing  adequate  oonoeptual 
coverage  of  the  data  base  and  its  contents,  the  activities  that  the  data  base 
describes,  and  the  dialogue  between  the  user  and  system,  the  implementation  of 
JEtS,  on  the  other  hand,  centers  on  capturing  a  degree  of  closure  in  the 
manipulation  of  concepts.  Closure,  as  defined  by  Woods  [1977],  implies 
handling  user  utterances  which  are  consistent  with  the  domain,  but  which  may 
not  have  been  specifically  foreseen.  Closure  can  be  approached  by  basing 
interpretation  on  rules  that  are  as  general  as  possible. 

The  conceptual  component  of  JETS  is  embedded  in  FRL  [Goldstein  and 
Roberts,  1977],  a  language  for  building  frame  based  systems.  The  knowledge  of 
JETS  is  organized  into  a  network  of  frames.  There  are  many  kinds  of  links 
associating  frames  with  one  another,  but  two  of  the  most  useful  are  the 
generality/specificity  links.  With  these  links,  the  conceptual  system  can  be 
thought  of  as  a  directed  tree  of  frames  rooted  at  the  most  general  concept, 
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THING.  The  more  specific  frames,  those  closer  to  the  leaves  of  the  tree, 

inherit  the  information  stored  with  tl.uir  ancestor  frames  and  add  some  more,, 
specific  information  to  that.  The  fundamental  advantage  of  the 

generality/specificity  hierarchy  is  that  the  decomposition  of  specific 
concepts  into  more  general  ones  promotes  the  identification  of  regularities 
among  the  concepts.  The  hierarchy  encourages  the  use  of  interpretation  rules 
that  are  written  to  apply  to  the  most  general  frames  possible,  and  then 

automatically  apply  to  their  more  specific  descendent s. 

Use  of  the  frames  in  JETS  will  be  made  primarily  by  a  set  of 
interpretation  rules.  The  rules  currently  consist  of  a  pattern  part  and  an 
action  part  with  procedures  attached  for  specifying  arbitrary  conditions. 
Implementing  the  interpretation  component  as  a  set  of  rules  facilitates  the 
insertion  and  deletion  of  rules.  This  means  that  incremental  changes  to  the 
rule  set  may  be  made  without  necessitating  the  understanding  of  elaborate 
interpretation  procedures. 

The  rules  are  organized  hier archicslly  by  generality,  the  more  general 
rules  being  closer  to  the  root.  The  hierarchical  organization  of  rules  is 
useful  in  resolving  problems  of  contention.  When  more  that  one  rule  qualifies 
to  be  applied  in  a  given  situation,  and  one  rule  is  a  descendent  of  another, 
the  more  specific  rule  is  chosen.  In  addition,  the  h ■ erarohical  organization 
of  rules  helps  identify  regularities  among  the  rules.  Regularities  among  the 
interpretation  rules  promote  the  writing  of  more  general  rules,  thus 
contributing  to  gaining  closure.  Thus,  we  see  several  points  which  should 
prepare  JETS  to  handle  the  problem  of  scale,  one  of  the  most  important 
problems  of  natural  language  systems:  the  ready  ability  to  make  incremental 
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changes  to  interpretation  rules,  the  concept  clustering  advantages  of  frames, 
and  the  identification  of  regularities  among  both  the  concept  frames  and  the 
interpretation  rules. 

Achieving  s laaafclc.  Cloaure 

A  part  of  our  work  is  oentered  on  achieving  closure  in  the  area  of 
semantic  Interpretation.  Our  focus  is  an  attempt  to  adequately  cover  simple 
noun  modification,  in  particular,  the  problem  of  interpreting  noun-noun 
modification. 

lhe  problem  of  interpreting  strings  of  nouns  related  through  modification 
is  a  complex  one.  For  our  purposes,  we  divide  the  problem  into  three 
subproblems:  lexical  interpretation,  modifier  parsing  and  oonoapt 
modification. 

By  lexical  interpretation  we  mean  the  process  of  mapping  the  lexioal 
items  (in  this  case  the  nouns  in  the  string)  into  appropriate  concepts.  The 
principal  difficulty  here  is  handling  words  with  multiple  senses. 

Modifier  parsing  is  the  process  of  discovering  the  internal  structure 
associated  with  the  string  of  nouns  or  the  concepts  which  result  after  lexical 
interpretation.  For  example,  a  string  of  three  nouns,  N1  N2  N3,  might  have 
the  structure  ((N1  N2)  W3 )  or  the  structure  (N1  (N2  N3)).  The  first  structure 
would  be  chosen  for  the  string  "engine  damage  reports"  and  the  second' for  the 
string  "replacement  oil  pump" . 

The  term  conceptual  modification  refers  to  the  problem  of  assigning  an 
interpretation  to  an  instance  of  one  concept  modifying  anotner  concept,  for 
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example,  when  the  ENGINE  concept  modifies  the  DAMAGE  concept  in  the  phrase 
"engine  damage"  we  wa^t  to  fill  the  object  role  in  the  DAMAGE  concept  with  the 
ENGINE  concept.  A  more  complex  example  is  the  interpretation  of  the  phrase  r. 
"engine  housing  acid  damage".  Here,  the  desired  result  is  a  something  like 
the  network  of  frames  shown  in  figure  [damage].  A  prose  description  of  this 
network  would  be: 

A  RESULT  of  a  DAMAGE  event  where  the  CAUSE  is  a 
CORROSION  event  in  which  the  AGENT  is  an  ACID  and 
the  object  is  the  HOUSING  part  of  an  ENGINE. 

These  three  subproblems  are,  of  course,  interrelated  and  cannot  be 
completely  decoupled.  In  our  initial  research  we  are  concentrating  on  the 
problem  of  conceptual  modification.  Our  goal  is  that ,  given  any  two  ooncepts 
that  are  cor.ectly  interpreted  by  the  system,  their  combination  (i.e.  through 
modification)  will  be  correctly  interpreted  by  the  system.  Note  that  the 
correct  interpretation  of  a  concept  does  not  imply  that  the  system  should  be 

able  to  "handle"  the  concept  in  the  sense  of  answering  questions  about  the 

/ 

concept  or  relating  it  to  the  data  base. 


The  problem  of  interpreting  noun-noun  modification  brings  the  issue  of 
closure  into  focus.  The  essential  feature  of  noun-noun  modification  is  that 
the  semantic  relationship  which  exists  between  the  two  nouns  is  not  explicit 
in  the  utterance.  Moreover,  a  large  number  of  relationships  may,  in 
principle,  be  possible  between  the  two  concepts  represented  by  the  nouns.  It 
is  the  responsibility  of  the  system  to  attempt  to  infer  or  discover  an 
appropriate  relationship,  given  its  understanding  of  the  two  concepts 
involved,  general  pragmatic  Knowledge,  and  the  current  discourse  context. 


8 


February  8  1979 


8 


Achieving  Completeness  Through  Coverage  and  Closure 


RESULT2 

•  •  • 

EVEN"'  *  DAMAGEM8 


0AMAQEh8 


CAUSE 

AGENT 

OBJECT 


CORROSION 16 
ACID3 
HOUSING 12 


C0RRS0SIQN16 
«  •  • 

AGENT  «  ACID3 

OBJECT  a  HOUSING 12 

RESULT  a  RESULT2 


ACID3 


HOUSING 12 

PARTOF  a  ENGINE6 


ENGINE6 


"engine  housing  acid  damage" 
Figure  [damage] 
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An  example;  Time 

As  an  example,  consider  the  use  of  a  time  phrase  used  to  modify  a  noun, 

as  in  the  phrases: 

January  Skyhawks  repairs 
1976  flights 

If  the  system  can  interpret  phrases  referring  to  time  (as  almost  any  system 
must)  then  it  should  attempt  to  interpret  the  modification  of  any  other 
concept  whioh  could  conceivably  have  a  time  phrase  attached  to  it.  It  should 
do  this  even  when  the  partiouler  domain  in  which  the  system  is  applied  does 
not  support  such  an  attachment. 

In  our  semantics,  a  time  phrase  can  only  be  used  to  modify  a  oonoept 
which  is,  or  can  be  viewed  as,  a  kind  of  an  EVENT.  A  minimal  amount  of 
closure  is  achieved  when  any  event  or  event  related  concept  oan  be 
successfully  modified  by  a  TIME  concept.  What  if  the  modified  oonoept  is  not 
an  EVENT  but  something  else,  say  an  OBJECT?  If  a  time  phrase  is  hypothesised 
to  modify  something  which  is  a  kind  of  OBJECT,  our  system  will  attempt  to 
derive  an  underlying  event  associated  with  that  object  to  attach  the  time 
phrase  to.  For  example,  in  the  standard  PARTS-SUPPLIERS-PROJECTS  domain,  the 
phrase : 

January  parts 

might  suggest  the  interpretations: 

parts  which  were  shipped  in  January 
parts  which  were  received  in  January 
parts  which  were  ordered  in  January 

In  such  an  impoverished  domain  this  is  almost  trivial,  as  one  can  precompute 
the  set  of  events  in  which  a  concept  can  partake. 
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In  s  semantically  rich  domain,  such  as  our  3-M  data  base  [NALDA,  1976], 
the  problem  is  much  more  difficult.  One  can  not  (or  perhaps  should  not) 
always  enumerate  the  potential  relationships  which  might  exist  between  even 
two  simple  concepts.  The  ability  to  handle  references  to  entities  and 
relations  mentioned  earlier  in  the  discourse  makes  the  problem  even  more 
complex.  This  allows  for  mre  potential  relationships  between  any  two 
conoepts.  For  example,  the  phrase 
the  January  planes 

could  be  used  to  refer  to  a  set  of  planes  introduced  previously  in  the 
discourse.  The  successful  Interpretation  of  this  phrase  would  require  a 
search  through  the  recent  discourse  to  discover  a  set  of  planes  which  was 
involved  in  an  event  which  occurred  in  January.  For  example,  the  context 
might  have  been  the  one  shown  in  figure  [context].  In  this  case,  "the  January 
planes"  should  be  interpreted  as  referring  to  "the  planes  which  received 
engine  maintenance  in  January  1976". 

Aa  example:  &eJm 

We  have  introduced  our  JETS  system  to  the  concept  of  a  SET.  In  order  to 
achieve  a  high  degree  of  semantic  closure,  the  system  should  be  able  to  form 
the  concept  of  a  SET  over  a  wide  domain  of  objects.  Our  previous  system, 
PLANES,  handled  sets  in  an  unsatisfactory  way.  One  could  only  refer  to  sets 
of  objects  of  a  certain  type  (e.g.  a  set  of  planes  or  maintenance  codes,  but 
not  a  set  of  parts  or  "how  malfunctioned"  codes).  Given  that  sets  can  be 
represented  and  formed  in  a  uniform  way  over  the  widest  possible  domain,  we 
must  turn  our  attention  to  issues  of  interpreting  the  modification  of  the  set 
concept. 
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<user> 

<JETS> 


Show  me  the  engine  maintenance  performed  on  F4*s  in 
the  last  three  months. 


DATE 

PLANE 

1/2/78 

3 

1/10/78 

3 

1/12/78 

23 

1/20/78 

23 

1/26/78 

3 

1/28/78 

43 

2/6/78 

4 

2/10/78 

32 

2/10/78 

23 

mmmm  sm 


A  discourse  context 
Figure  [context] 
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The  ability  to  form  sets  of  arbitrary  elements  will  be  of  little  use  if 
the  semantic  interpretation  rules  do  not  allow  one  to  modify  such  sets  in  -a- 
general  way.  Thus,  if  the  system  knows  what  it  means  for  concept  X  to  modify' 
concept  X,  then  it  should  know  what  it  means  for  concept  X  to  modify  a  concept 
Z  where  Z  is  a  set  whose  members  are  concepts  which  are  X's. 

Our  approach  is  to  include  a  meta-rule  for  sets  which  uses  rules 
applicable  to  the  particular  domain  of  a  set.  Our  SET  frame  has  a  slot  for  a 
typical  member  as  well  as  one  to  receive  the  actual  members,  if  there  are  any. 
This  slot  refers  to  a  frame  which  describes  the  typical  member  of  the  set. 
Whenever  we  wish  to  modify  a  set  by  another  concept,  this  meta-rule  will 
search  for  primitive  rules  which  interpret  modification  of  the  set's  typical 
member  by  that  concept.  The  rules  whioh  are  found  to  be  applicable  are  then 
invoked  on  the  typical  member  and  to  the  set's  individual  members,  if  any 
exist.  This  meta-rule  for  sets  is  shown  in  figure  [meta-rulel ] . 

Let's  examine  the  use  of  this  meta-rule  for  sets  in  the  interpretation  of 
the  phrase: 

Planes  3,  5,  and  48 

At  the  concept  level,  this  phrase  is  represented  as  the  general  PLANE  concept 
modifying  a  SET  concept.  This  particular  SET  has  the  INTEGER  concept  for  its 
typical  member  and,  as  individual  members,  the  concepts  for  the  integer  3,  the 
integer  5,  and  the  integer  48. 

One  of  the  rule-:,  applicable  when  a  PLANE  modifies  an  INTEGER,  is  a  rule 
which  interprets  the  integer  as  representing  the  plane's  serial  number.  In 
cur  world,  the  only  planes  which  have  serial  numbers  are  those  in  the  data 
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if  <concept>  modifies  a  <set>  then: 

find  the  typical  member  of  the  <set>. 

find  an  applicable  rule  which  interprets  the  . 
modification  of  typical  member  by  <concept>. 

invoke  the  rule  on  the  typical  member. 

invoke  the  rule  on  each  of  the  members  of  the 
<set>. 

return  the  newly  modified  <set>. 


An  interpretation  rule  for  sets 
figure  [meta-rulel] 
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base.  These  planes  are  represented  by  the  more  specific  concept  3M-PLANE> 
The  action  of  this  rule'  is  to  view  (*)  the  plane  concept  as  a  3M-PLANE  and'  the 
integer  concept  as  a  SERIAL  NUMBER.  The  3M-PLANE's  serial  number  slot  is  then, 
filled  with  the  SERIAL-NUMBER  concept. 

When  this  rule  is  found  by  the  meta-rule  for  sets,  it  is  applied  to  both 
the  set’s  typical-member  filler  (in  this  case  the  generic  INTEGER  concept)  and 
to  the  set’s  members  (which  are  the  integers  3,  5  and  48).  The  result  of  all 
this  is  described  by  the  concept  frame: 

A  SET  with 

typical  member  s  a  3M-PLANE 
members  s  a  3M-PLANE  with 

serial  number  *  an  INTEGER  with  value  s  3 
a  3M-PLANE  with 

serial  number  s  an  INTEGER  with  value  *  5 
a  3M-PLANE  with 

serial  number  s  an  INTEGER  with  value  a  48 

To  handle  the  case  where  a  SET  is  used  to  modify  another  concept,  we 
include  another  meta-rule,  shown  in  figure  [meta-rule2] .  Consider  the  role  of 
this  rule  in  the  interpretation  of  the  phrase: 

radar  and  navigation  equipment  failures 

The  phrase  "radar  and  navigation  equipment"  is  interpreted  as  a  SET  whose 
typical  member  is  a  FUNCTIONAL-PLANE-SUBSYSTEM  and  whose  members  are  the 
concepts  RADAR-SUBSYSTEM  and  NAVIGATION-SUBSYSTEM.  In  interpreting  the 


*  Viewing  a  concept  X  as  a  concept  Y  is  a  process  which  maps  the 
information  in  X  into  a  newly  instantiated  Y  concept.  If  X  is  a  kind  of  Y,  no 
mapping  need  be  done,  however. 
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if  a  <set>  modifies  a  <concept>  then; 

find  the  tvpioal  member  of  the  <set> . 

find  .a  rule  which  interprets  the  modification  of 
the  <concept>  by  the  tvpioal  member. 

form  a  new  instantiation  of  a  SET. 

fill  the  new  typical  member  slot  by  invoking 
the  rule  on  the  tvpioal  member  and  <concept>. 

fill  the  members  of  the  new  SET  by  invoking  the 
rule  on  the  member  of  the  old  SET  and  the  old 
<concept>. 

.ieturn  the  new  SET. 


An  interpretation  rule  for  sets 
figure  [meta-rule2] 
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modification  of  the  FAILURE  concept  by  this  SET,  the  meta-rule  for  sets  £s; 

•  ’  'r 

invoked  and  attempts  to  find  rules  which  guide  the  interpretation  of  a  kind  of 
FUNCTIONAL-PLANE-SUBSYSTEM  modifying  FAILURE.  The  rule  which  is  most 
applicable  is  one  which  interprets  the  modifying  concept  (the  subsystem)  as 
filling  the  failure  location  role  in  the  FAILURE  concept.  The  final 
interpretation  of  this  phrase  results  in  a  SET  of  FAILURES  in  which  the' 
typical  member  is  a  failure  in  a  FUNCTIONAL-PLANE-SUBSYSTEM  and  which  contains 
two  members:  a  FAILURE  in  the  RADAR-SUBSYSTEM  and  a  failure  in  the  NAVIGATION- 
SUBSYSTEM. 

Pr-QbLemr.5Qly.iBft  Luamaa. 

The  discussion  so  far  has  centered  around  the  need  for  achieving  closure 
and  possible  ways  of  accomplishing  it  in  JETS.  However,  nothing  has  been  done 
to  help  the  system  develop  a  plan  to  extract  the  required  information  from  the 
data  base  in  the  form  the  user  intended.  This  is  where  problem-solving  frames 
come  to  the  rescue.  Problem-solving  frames  are  to  be  used  to  describe  problem 
domains  and  search  strategies.  They  can  range  from  very  general  sketches  of  a 
particular  series  of  problem  environments  to  specific  suggestions  on  how  to 
answer  a  certain  request.  The  last  type  of  problem-solving  frame  might  even 
be  encoded  in  English— where  a  sequence  of  English  instructions  are  given  on 
how  to  put  all  of  the  information  together  at  the  end  to  answer  the  main 
request.  Such  frames  will  have  to  work  closely  with  the  frames  used  during 
the  parsing  and  semantic  analysis  stages  so  that  appropriate  values  can  be 
found  to  fill  in  empty  slots  in  the  problem-solving  frames  in  order  that  the 
whole  frame  may  become  instantiated. 
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HattsDefinefl  £cafclaa 

Before  describing  the  details  of  problem-solving  frames,  a  description  of 
what  a  "problem"  is  should  be  discussed.  A  well-defined  problem  has  been 
defined  in  the  literature  to  consist  of  the  following  properties: 

1)  expressed  in  terms  the  solver  can  understand, 

2)  all  information  necessary  for  solving  the 
problem  should  be  in  the  problem  statement, 

3)  the  form  of  the  solution  is  exactly 
specified,  and 

4)  there  must  be  a  systematic  (algorithmic) 
way(s)  for  testing  a  proposed  solution. 

In  our  natural  language  query  system  environment  (in  particular  with  our 
experience  with  PLANES)  we  find  that  many  requests  by  users  are  not  normally 
well-defined,  i.e.  they  do  not  satisfy  the  criteria  above.  Before  such 
requests  can  be  handled,  it  is  necessary  to  make  the  request  a  well-defined 
problem.  Previous  systems  have  brought  in  this  additional  knowledge  primarily 
through  two  sources— using  past  context  to  fill  in  missing  information  and/or 
asking  the  user  directly  for  it.  We  want  to  introduce  the  use  of  world 
knowledge  as  a  third  way  of  obtaining  such  information.  At  the  same  time  we 
want  to  use  the  world  knowledge  to  detect  requests  not  answerable  with  data 
available  in  the  data  base. 

gUEMae  a£  Prob),$m-SglYinK  F.ramss 

Collecting  together  such  information  still  does  little  for  bringing  the 
system  closer  to  developing  a  plan  for  answering  the  request.  The  problem¬ 
solving  frames  are  to  propose  general  techniques- -such  as  problem  reduction 
(reducing  a  problem  to  simpler  subproblems)— for  solving  problems.  To  be  able 
to  develop  a  set  cf  problem-solving  frames  that  classify  problems  and 
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techniques  for  solving  them,  we  must  categorize  the  "kinds"  of  problems 
encountered  in  our  data  base  environment  (drawing  on  the  fact  that  the  data 
base  world  restricts  us  enough  that  the  number  of  interesting  classes  of 
problems  is  manageable).  Our  studies  have  shown  that  the  following  kinds  of 
requests  occur  most  often:  statistical  analyses  (correlation  of  data,  etc.), 
association  of  data  types  to  each  other,  ranking  ("top  five",  etc.),  plotting, 
causality  (generalize  concept),  very  general  inferencing  and  operations  on 
sets. 


Another  major  contribution  would  be  the  ability  to  provide  different 
"views"  to  a  problem  and  to  the  values  stored  in  the  data  base.  In  essence  an 
"overlay"  of  the  data  base  could  be  generated  for  a  particular  problem.  An 
example  would  be  data  stored  in  a  daily  format  in  the  data  base  viewed  as  if 
weekly  data  existed.  This  same  mechanism  could  be  applied  to  bring  two 
seemingly  disjoint  concepts  in  a  problem  together. 

A  Sfi.enarl.a 

A  sample  scenario  of  the  use  of  problem-solving  frames  can  be  seen  in 
Figure  [scenar].  Certain  frames  are  activated  by  key  words  and  phrases  in  the 
user's  request.  The  activated  frames  analyze  the  request  to  decide  if  they 
can  provide  any  guidance  in  formulating  a  plan  capable  of  rendering  a  solution 
to  the  problem.  If  no  such  assistance  can  be  offered,  the  frame  is 
deactivated;  otherwise  complete  control  is  passed  to  that  problem-solving 
frame.  Figure  [compar]  gives  an  example  of  a  problem-solving  frame  for 
compare.  Notice  the  use  of  production-like  rules  that  associate  specific 
conditions  and  actions.  These  actions  include  drawing  in  other  problem¬ 
solving  frames  and  invoking  specific  functions  such  as  MAKfci-UNITS-EQOAL  which 
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<user> 

<jets> 


<user> 

<jets> 


<user> 
< jets> 


A  Scenario 

Compare  NORS  and  NORMUS  percentages  by  squadron  and  by 
wing  for  May  73. 

Invoking  COMPARE  frame. 

Invoking  PERCENT  frame. 

NOR  (Not  Operationally  Ready)  does  not  exist  in 
the  data  base.  It  can  be  obtained  by  ADDING 
NORS  (Not  Operationally  Ready  due  to  Scheduled 
Maintenance)  and  NORMUS  (Not  Operationally  Ready 
due  to  Unscheduled  Maintenance) . 

SQUADRON  is  a  classification  that  partitions  the 
existing  planes  up.  Unfortunately,  no  such  data 
is  stored  in  the  data  base.  Such  partitioning  can 
be  done  by: 

(1)  plane  serial  number 

(2)  plane  type 

(3)  none  of  the  above 
Please  select  one  of  the  above: 

2 

WING  linked  to  SQUADRON. 

PLANE  TYPE  absorbing  WING. 

I  have  interpreted  your  request  as  follows: 

1.  Retrieve  NORS,  NORMUS  by  plane  types  for 
time  period  of  May  1973. 

2.  Form  NORsNORS+NORMUS  by  plane  types  for 
time  period  of  May  1973. 

3.  Generate  NORS  percentages  NORS/NOR  and 
NORMUS  percentagesNORMUS/NOR  by  plane  type 
for  time  period  of  May  1973. 

4.  Construct  table  of  PLANE  TYPE,  NORS  percentage, 
NORMUS  percentage. 

5.  List  by  PLANE  TYPE  the  maximum  of  NORS  percentage 
or  NORMUS  percentage. 

Does  this  satisfy  your  request? 

yes 

Executing. . . 
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PLANE  TYPE 

NORS? 

NORMUS? 

A7 

71# 

29? 

F4 

55? 

45? 

SKYHAWKS 

27? 

73? 

Interpreting. . 

• 

A7 :  more  NOr.S 

f 4 :  more  NORs 

SKYHAWKS:  more 

NORMUS 

Figure  [scenar] 
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COMPARE  Frame 


Invoking  Concept:  COMPARE,  MORE,  LESS,  ... 

Context(s):  Requires  instantiation  of: 

(1)  one  FIELD  over  two  or  more  ranges, 
or  (2)  two  or  more  different  FIELDs:  FIELD1 , 
FIELD2 , . . . 

Actions : 

Contexts (1 ) : 

IF  FIELD. TYPEsnum  &  KEYWORDsmore 
THEN  invoke  greater 
IF  FIELD. TYPEsr.ura  &  KEYWORDsless 
THEN  invoke  less 


Contexts (2) : 

IF  FIELD1.TYPEsn.ura  &  FIELD2  .TYPE=nnum 
&  KEYWORDsmore 
THEN  invoke  greater 
IF  FIELD1. TYPEsnum  i  FIELD2 .TYPEs nnum 
&  KEYWORDsless 

THEN  invoke  less 

IF  FIELDl.TYPEsnum  4  FIELD2 .TYPEsset  &  KEYWORDsmore 
THEN  invoke  cardinality-of-set 
invoke  mo^e 

IF  FIELDl.TYPEsnum  &  FIELD2 .TYPEsnum 
&  KEYWORDscompare 
THEN  invoke  find-relationship 
IF  KEYWORDsassociate 

THEN  invoke  check-correlation 
invoke  find-relaticnship 


Global  Actions:  make-uni ts-equal 


Figure  [compar] 
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tries  to  convert  the  units  of  measurement  (time,  length,  etc.)  of  the 
individual  fields  into  equivalent  forms.  Figure  [units]  demonstrates  this 
concept  by  exhibiting  the  hierarchy  of  unit  frames. 

In  summary,  the  value  of  this  type  of  frame  is  that  it  provides  a  general 
forum  for  taking  all  the  information  provided  by  the  instantiated  frames  and 
analyzing  it  so  that  a  program  can  be  generated  to  answer  the  question.  In 
other  words,  it  is  not  responsible  for  writing  a  program  at  the  formal  query 
level  but  only  to  decide  what  question  to  answer,  to  break  the  question  up 
into  a  sequence  of  simpler  requests  if  the  question  is  too  complex,  to  provide 
higher  level  raathematical/statistical  analysis  of  the  data  returned,  and  to 
call  on  a  formal  query  generator  to  generate  the  actual  formal  query. 

Usmlliaft  lagus.  iici  Coiaaisx  B&gmea&a 

The  other  type  of  problem-solving  frame  not  yet  described  is  to  be  used 
in  analyzing  vague  and  complex  questions.  Each  frame  will  consist  in  essence 
of  a  sequence  of  simpler  commands,  where  each  simpler  command  is  either  the 
sort  of  question  which  can  be  understood  directly,  or  another  command  which 
can  ultimately  be  broken  up  into  a  sequence  of  questions  that  can  be  answered. 
They  will  also  be  used  to  generate  dialogue  with  the  user  when  vague  terms 
must  be  further  explained  before  a  formal  query  could  be  generated  (e.g. 
defining  "worst"  for  the  particular  user  and  question) . 

In  JETS,  this  kind  of  problem-solving  frame  is  useful  for  report 
generation.  Tue  following  describes  what  a  problem-solving  frame  for  requests 
like  "Does  trend  analysis  of  failure  and  maintenance  rates  differ 
significantly  from  the  corresponding  rates  of  new  aircraft?"  would  do. 


2? 


February  3  1979 


23 


Achieving  Completeness  Through  Coverage  and  Closure 


Make-Uni ts-Equal 


UNITS 


time  distance 


HOUR  DAY  WEEK...  METER  INCH  ... 


Figure  [units] 
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(1)  define  "differ  significantly''  ( user^defined  or 
system  default), 

(2)  retrieve  for  each  maintenance  action  X  all  relevant 
data  cards  (i.e.  the  "begin"  card  and  "end"  card), 

(3)  for  each  maintenance  action  form  maintenance  rate 
(endtime  -  begintime) , 

(4)  project  trend  of  maintenances  rates  by  linear 
regression, 

(5)  calculate  average  mean  time  between  failure  (this 
might  invoke  another  problem-solving  frame), 

(6)  apply  (1)  through  (5)  to  new  aircraft, 

(7)  generate  table  of  trend  of  failure  rate  and 
maintenance  rate  for  all  data  vs.  trend  of  failure 
rate  and  maintenance  rate  for  new  aircraft,  and 

(8)  for  trena  of  failure  rate  and  maintenance  rate  for 
all  data  and  new  aircraft  compute  standard  deviations. 


The  problem-solving  frame  writes  all  of  the  program  required  except  the 
generation  of  the  basic  formal  queries  (the  ones  that  retrieve  the  actual 
fields  used  to  calculate  the  failure  and  maintenance  rates)  which  are 
generated  by  the  formal  query  generator. 
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